Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages, which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission if necessary.

Note: Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

In addition to implementing code, there is a writeup to complete. The writeup should be completed in a separate file, which can be either a markdown file or a pdf document. There is a write up template that can be used to guide the writing process. Completing the code template and writeup template will cover all of the rubric points for this project.

The rubric contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this Ipython notebook and also discuss the results in the writeup file.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 0: Load The Data

In this module, we need to initialize and import the train_data, valid_data and test_data Using panda is easily to access the csv file.

In [1]:
# Load pickled data
import cv2
import pickle
import numpy as np
from matplotlib import pyplot
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
from pandas.io.parsers import read_csv
from skimage import transform as transf



# TODO: Fill this in based on where you saved the training and testing data

training_file = 'dataset/train.p'
validation_file='dataset/valid.p'
testing_file = 'dataset/test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
sign_names = read_csv("signnames.csv").values[:, 1]
    
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']

Step 1: Dataset Summary & Exploration

The pickled data is a dictionary with 4 key/value pairs:

  • 'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
  • 'labels' is a 1D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
  • 'sizes' is a list containing tuples, (width, height) representing the original width and height the image.
  • 'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below. Use python, numpy and/or pandas methods to calculate the data summary rather than hard coding the results. For example, the pandas shape method might be useful for calculating some of the summary results.

Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas

In this modul, we will get the shape of train_data, valid_data and test_data .

In [2]:
### Replace each question mark with the appropriate value. 
### Use python, pandas or numpy methods rather than hard coding the results

# TODO: Number of training examples
n_train = X_train.shape[0]

# TODO: Number of validation examples
n_validation = X_valid.shape[0]

# TODO: Number of testing examples.
n_test = X_test.shape[0]

# TODO: What's the shape of an traffic sign image?

image_shape = X_train[0].shape

# TODO: How many unique classes/labels there are in the dataset.
# By using the np.unique we can get the numbers , indices and counts of different classes in original data.
classes, class_indices, class_counts  = np.unique(y_train, return_index=True, return_counts=True)
n_classes = len(classes)

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 34799
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43
In [3]:
# By using plt we can draw the histogram of class distribution of data
plt.figure(0, figsize = (10,5))
train_classes, train_counts = np.unique(y_train, return_counts = True)
plt.bar(train_classes, train_counts)

plt.title('Training Set Class Distribution', fontsize=22)
plt.xlabel('Class Number', fontsize=20)
plt.ylabel('Number of Occurances', fontsize=20)
plt.tick_params(labelsize=16)
plt.grid(linestyle=':')

Include an exploratory visualization of the dataset

Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.

The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.

NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections. It can be interesting to look at the distribution of classes in the training, validation and test set. Is the distribution the same? Are there more examples of some classes than others?

In [4]:
### Data exploration visualization code goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline

for each_c, each_c_i, each_c_c in zip(classes, class_indices, class_counts):
    print(each_c, ". Class : ", sign_names[each_c] )
    figs,axes=plt.subplots(1,10,figsize=(10,1))
    figs.subplots_adjust(left = 0, right = 1, bottom = 0, top = 1, hspace = 0.05, wspace = 0.05)
    for i in range(10):
        random_indices = np.random.randint(each_c_i, each_c_i+each_c_c, 10)
        axes[i].imshow(X_train[random_indices[i],:,:,:]) 
        axes[i].axis('off')        
    pyplot.show()
0 . Class :  Speed limit (20km/h)
1 . Class :  Speed limit (30km/h)
2 . Class :  Speed limit (50km/h)
3 . Class :  Speed limit (60km/h)
4 . Class :  Speed limit (70km/h)
5 . Class :  Speed limit (80km/h)
6 . Class :  End of speed limit (80km/h)
7 . Class :  Speed limit (100km/h)
8 . Class :  Speed limit (120km/h)
9 . Class :  No passing
10 . Class :  No passing for vehicles over 3.5 metric tons
11 . Class :  Right-of-way at the next intersection
12 . Class :  Priority road
13 . Class :  Yield
14 . Class :  Stop
15 . Class :  No vehicles
16 . Class :  Vehicles over 3.5 metric tons prohibited
17 . Class :  No entry
18 . Class :  General caution
19 . Class :  Dangerous curve to the left
20 . Class :  Dangerous curve to the right
21 . Class :  Double curve
22 . Class :  Bumpy road
23 . Class :  Slippery road
24 . Class :  Road narrows on the right
25 . Class :  Road work
26 . Class :  Traffic signals
27 . Class :  Pedestrians
28 . Class :  Children crossing
29 . Class :  Bicycles crossing
30 . Class :  Beware of ice/snow
31 . Class :  Wild animals crossing
32 . Class :  End of all speed and passing limits
33 . Class :  Turn right ahead
34 . Class :  Turn left ahead
35 . Class :  Ahead only
36 . Class :  Go straight or right
37 . Class :  Go straight or left
38 . Class :  Keep right
39 . Class :  Keep left
40 . Class :  Roundabout mandatory
41 . Class :  End of no passing
42 . Class :  End of no passing by vehicles over 3.5 metric tons

Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!

With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89. To meet specifications, the validation set accuracy will need to be at least 0.93. It is possible to get an even higher accuracy, but 0.93 is the minimum for a successful project submission.

There are various aspects to consider when thinking about this problem:

  • Neural network architecture (is the network over or underfitting?)
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

Pre-process the Data Set (normalization, grayscale, etc.)

Minimally, the image data should be normalized so that the data has mean zero and equal variance. For image data, (pixel - 128)/ 128 is a quick way to approximately normalize the data and can be used in this project.

Other pre-processing steps are optional. You can try different techniques to see if it improves performance.

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project.

In this modul, we will perform data augmentation. It include rotation, translation, scaling, shear and light-change operation.

In [5]:
### Preprocess the data here. It is required to normalize the data. Other preprocessing steps could include 
### converting to grayscale, etc.
### Feel free to use as many code cells as needed.


def random_transform(img,angle_range=[-10,10],
                    scale_range=[0.8,1.2],
                    translation_range=[-5,5],
                    shear_range=[-0.2,0.2]):
    # Generate random parameter values
    rotated_img = 255*transf.rotate(img, angle=np.random.uniform(angle_range[0], angle_range[1]), mode='edge')
    translated_img = transf.warp(rotated_img, transf.SimilarityTransform(translation=(np.random.uniform(translation_range[0], translation_range[1]), np.random.uniform(translation_range[0], translation_range[1]))),mode='edge')
    scaled_img = transf.warp(translated_img,
                  transf.AffineTransform(scale=(np.random.uniform(scale_range[0], scale_range[1]), np.random.uniform(scale_range[0],scale_range[1])), shear=np.random.uniform(shear_range[0],shear_range[1])),
                  mode='edge')

    return scaled_img.astype(np.uint8)

def random_brightness(image,min_bright=0.25,max_bright=1):
    image = image.astype(np.uint8)
    image1 = cv2.cvtColor(image,cv2.COLOR_RGB2HSV)
    random_bright = np.random.uniform(min_bright, max_bright)
    image1[:,:,2] = image1[:,:,2]*random_bright
    image1 = cv2.cvtColor(image1,cv2.COLOR_HSV2RGB)
    return image1
In [6]:
# Combine all data augmentation methods.
def transform_image(image):
    result=random_transform(random_brightness(image)).astype(np.uint8)
    return  result
    
In [7]:
# visualization of results of data augmentation
for each_c, each_c_i, each_c_c in zip(classes, class_indices, class_counts):
    print(each_c, ". Class : ", sign_names[each_c] )
    figs,axes=plt.subplots(1,10,figsize=(10,1))
    figs.subplots_adjust(left = 0, right = 1, bottom = 0, top = 1, hspace = 0.05, wspace = 0.05)
    for i in range(10):
        random_indices = np.random.randint(each_c_i, each_c_i+each_c_c, 10)
        axes[i].imshow(transform_image(X_train[random_indices[i],:,:,:]))
        axes[i].axis('off')        
    pyplot.show()
0 . Class :  Speed limit (20km/h)
1 . Class :  Speed limit (30km/h)
2 . Class :  Speed limit (50km/h)
3 . Class :  Speed limit (60km/h)
4 . Class :  Speed limit (70km/h)
5 . Class :  Speed limit (80km/h)
6 . Class :  End of speed limit (80km/h)
7 . Class :  Speed limit (100km/h)
8 . Class :  Speed limit (120km/h)
9 . Class :  No passing
10 . Class :  No passing for vehicles over 3.5 metric tons
11 . Class :  Right-of-way at the next intersection
12 . Class :  Priority road
13 . Class :  Yield
14 . Class :  Stop
15 . Class :  No vehicles
16 . Class :  Vehicles over 3.5 metric tons prohibited
17 . Class :  No entry
18 . Class :  General caution
19 . Class :  Dangerous curve to the left
20 . Class :  Dangerous curve to the right
21 . Class :  Double curve
22 . Class :  Bumpy road
23 . Class :  Slippery road
24 . Class :  Road narrows on the right
25 . Class :  Road work
26 . Class :  Traffic signals
27 . Class :  Pedestrians
28 . Class :  Children crossing
29 . Class :  Bicycles crossing
30 . Class :  Beware of ice/snow
31 . Class :  Wild animals crossing
32 . Class :  End of all speed and passing limits
33 . Class :  Turn right ahead
34 . Class :  Turn left ahead
35 . Class :  Ahead only
36 . Class :  Go straight or right
37 . Class :  Go straight or left
38 . Class :  Keep right
39 . Class :  Keep left
40 . Class :  Roundabout mandatory
41 . Class :  End of no passing
42 . Class :  End of no passing by vehicles over 3.5 metric tons
In [8]:
# Perform data augmentation
# First: random transform images
# Seconde: generate more transformed images to make the distribution of classes same.

def data_augmentation(X_train, y_train, total_per_class =5000):
    classes, class_indices, class_counts  = np.unique(y_train, return_index=True, return_counts=True)
    height, width, channels = X_train[0].shape
    X_total = np.empty([0, X_train.shape[1], X_train.shape[2], X_train.shape[3]], dtype = np.float32)
    y_total = np.empty([0], dtype = y_train.dtype)
    
    for each_c,each_c_i, each_c_c in zip(classes,class_indices, class_counts):
        num_extra=total_per_class-each_c_c
        ##Copy over the current data for the given class
        X_orig = X_train[y_train == each_c]
        y_orig = y_train[y_train == each_c]
        ##Add original data to the new dataset
        X_total = np.append(X_total, X_orig, axis=0)
        y_total=np.append(y_total,y_orig)
        print(each_c," : ",each_c_c," + ", num_extra)
        X_extra = np.empty([num_extra, X_train.shape[1], X_train.shape[2], X_train.shape[3]], dtype = np.float32)
        y_extra=np.empty([num_extra],dtype = y_train.dtype)
        for i in range(num_extra):
            rand_id=np.random.randint(each_c_i,each_c_i+each_c_c)
            X_extra[i,:,:,:] = transform_image(X_train[rand_id]).reshape((1, height, width, channels))
            y_extra[i]=each_c
            
        X_total = np.append(X_total,X_extra, axis=0)
        y_total= np.append(y_total,y_extra)
        
    return X_total.astype(np.uint8), y_total
In [9]:
# Save the augmentation data
# I just generate max 4000 images, because large samples will slow the computation speed of my computer. You can set more bigger.
# Dont need run it every time, just one time
X_train_aug,y_train_aug=data_augmentation(X_train, y_train, total_per_class =4000)
np.savez_compressed('dataset/AugTrafficdata', X_train_aug = X_train_aug, y_train_aug = y_train_aug)
0  :  180  +  3820
1  :  1980  +  2020
2  :  2010  +  1990
3  :  1260  +  2740
4  :  1770  +  2230
5  :  1650  +  2350
6  :  360  +  3640
7  :  1290  +  2710
8  :  1260  +  2740
9  :  1320  +  2680
10  :  1800  +  2200
11  :  1170  +  2830
12  :  1890  +  2110
13  :  1920  +  2080
14  :  690  +  3310
15  :  540  +  3460
16  :  360  +  3640
17  :  990  +  3010
18  :  1080  +  2920
19  :  180  +  3820
20  :  300  +  3700
21  :  270  +  3730
22  :  330  +  3670
23  :  450  +  3550
24  :  240  +  3760
25  :  1350  +  2650
26  :  540  +  3460
27  :  210  +  3790
28  :  480  +  3520
29  :  240  +  3760
30  :  390  +  3610
31  :  690  +  3310
32  :  210  +  3790
33  :  599  +  3401
34  :  360  +  3640
35  :  1080  +  2920
36  :  330  +  3670
37  :  180  +  3820
38  :  1860  +  2140
39  :  270  +  3730
40  :  300  +  3700
41  :  210  +  3790
42  :  210  +  3790
In [10]:
# visualization of new data after data augmentation
plt.figure(0, figsize = (10,5))
total_classes, total_indices,total_counts = np.unique(y_train_aug,  return_index=True,return_counts = True)
plt.bar(total_classes, total_counts)

plt.title('Augmented Set Class Distribution', fontsize=22)
plt.xlabel('Class Number', fontsize=20)
plt.ylabel('Number of Occurances', fontsize=20)
plt.tick_params(labelsize=16)
plt.grid(linestyle=':')

for each_c, each_c_i, each_c_c in zip(total_classes, total_indices, total_counts):
    print(each_c, ". Class : ", sign_names[each_c] )
    figs,axes=plt.subplots(1,10,figsize=(10,1))
    figs.subplots_adjust(left = 0, right = 1, bottom = 0, top = 1, hspace = 0.05, wspace = 0.05)
    for i in range(10):
        random_indices = np.random.randint(each_c_i, each_c_i+each_c_c, 10)
        axes[i].imshow(X_train_aug[random_indices[i],:,:,:])
        axes[i].axis('off')        
    pyplot.show()
0 . Class :  Speed limit (20km/h)
1 . Class :  Speed limit (30km/h)
2 . Class :  Speed limit (50km/h)
3 . Class :  Speed limit (60km/h)
4 . Class :  Speed limit (70km/h)
5 . Class :  Speed limit (80km/h)
6 . Class :  End of speed limit (80km/h)
7 . Class :  Speed limit (100km/h)
8 . Class :  Speed limit (120km/h)
9 . Class :  No passing
10 . Class :  No passing for vehicles over 3.5 metric tons
11 . Class :  Right-of-way at the next intersection
12 . Class :  Priority road
13 . Class :  Yield
14 . Class :  Stop
15 . Class :  No vehicles
16 . Class :  Vehicles over 3.5 metric tons prohibited
17 . Class :  No entry
18 . Class :  General caution
19 . Class :  Dangerous curve to the left
20 . Class :  Dangerous curve to the right
21 . Class :  Double curve
22 . Class :  Bumpy road
23 . Class :  Slippery road
24 . Class :  Road narrows on the right
25 . Class :  Road work
26 . Class :  Traffic signals
27 . Class :  Pedestrians
28 . Class :  Children crossing
29 . Class :  Bicycles crossing
30 . Class :  Beware of ice/snow
31 . Class :  Wild animals crossing
32 . Class :  End of all speed and passing limits
33 . Class :  Turn right ahead
34 . Class :  Turn left ahead
35 . Class :  Ahead only
36 . Class :  Go straight or right
37 . Class :  Go straight or left
38 . Class :  Keep right
39 . Class :  Keep left
40 . Class :  Roundabout mandatory
41 . Class :  End of no passing
42 . Class :  End of no passing by vehicles over 3.5 metric tons
In [11]:
# From here we start to prepare and train deep learning model

from sklearn.utils import shuffle
import numpy as np
import tensorflow as tf
import os
import cv2
In [13]:
loaded = np.load('dataset/AugTrafficdata.npz')
X_train_aug = loaded['X_train_aug']
y_train_aug = loaded['y_train_aug']
print("Augmented Dataset size X")
print(X_train_aug.shape)
print("Augmented Dataset size y")
print(y_train_aug.shape)
Augmented Dataset size X
(172000, 32, 32, 3)
Augmented Dataset size y
(172000,)
In [14]:
plt.figure(0, figsize = (10,5))
valid_classes, valid_counts = np.unique(y_valid, return_counts = True)
plt.bar(valid_classes, valid_counts)

plt.title('Vaild Set Class Distribution', fontsize=22)
plt.xlabel('Class Number', fontsize=20)
plt.ylabel('Number of Occurances', fontsize=20)
plt.tick_params(labelsize=16)
plt.grid(linestyle=':')
In [15]:
plt.figure(0, figsize = (10,5))
test_classes, test_counts = np.unique(y_test, return_counts = True)
plt.bar(test_classes, test_counts)

plt.title('Test Set Class Distribution', fontsize=22)
plt.xlabel('Class Number', fontsize=20)
plt.ylabel('Number of Occurances', fontsize=20)
plt.tick_params(labelsize=16)
plt.grid(linestyle=':')
In [16]:
# Preprocessing the images by using histogram equalization
def equalize_histogram(image):
    image = image.astype(np.uint8)
    image_yuv = cv2.cvtColor(image, cv2.COLOR_BGR2YUV)
    
    # equalize the histogram of the Y channel
    clahe = cv2.createCLAHE(clipLimit=10.0, tileGridSize=(8,8))
    image_yuv[:,:,0] = clahe.apply(image_yuv[:,:,0])
    
    # convert the YUV image back to RGB format
    image_output = cv2.cvtColor(image_yuv, cv2.COLOR_YUV2BGR)
    return image_output

# Perform images normalisation 
def normalisation(X):
    X_n = np.empty(shape=X.shape)
    a = 0.1
    b = 0.9
    for i, img in enumerate(X):
        temp = equalize_histogram(img)
        X_max = np.amax(temp)
        X_min = np.amin(temp)
        X_n[i] = a + (temp - X_min)*(b - a)/(X_max - X_min)
    return X_n
In [17]:
#  visualization of new data after normalisation
test_image = X_train[np.random.randint(0, X_train.shape[0])]
test_image_eqhist = equalize_histogram(test_image)
test_image_norm = normalisation(np.expand_dims(test_image, axis=0))

f, axarr = plt.subplots(1,3)
axarr[0].imshow(test_image, interpolation='nearest')
axarr[0].set_title("Original image")
axarr[1].imshow(test_image_eqhist, interpolation='nearest')
axarr[1].set_title("+ Histogram" + '\n' + "equalisation")
axarr[2].imshow(test_image_norm[0], interpolation='nearest')
axarr[2].set_title("+ Normalisation")

plt.savefig('Normalisation.jpg', bbox_inches='tight')
plt.show()
In [18]:
# Preprocessing all data by using normalisation and equalization.

X_train = normalisation(X_train)
X_train_aug = normalisation(X_train_aug)
X_valid = normalisation(X_valid)
X_test=normalisation(X_test)

print(X_train.shape)
print(X_valid.shape)
print(X_test.shape)
print(X_train_aug.shape)
(34799, 32, 32, 3)
(4410, 32, 32, 3)
(12630, 32, 32, 3)
(172000, 32, 32, 3)
In [19]:
# Preprocessing all data by using normalisation and equalization.
aug_classes, aug_indices,aug_counts = np.unique(y_train_aug,  return_index=True,return_counts = True)

for each_c, each_c_i, each_c_c in zip(aug_classes, aug_indices, aug_counts):
    print(each_c, ". Class : ", sign_names[each_c] )
    figs,axes=plt.subplots(1,10,figsize=(10,1))
    figs.subplots_adjust(left = 0, right = 1, bottom = 0, top = 1, hspace = 0.05, wspace = 0.05)
    for i in range(10):
        random_indices = np.random.randint(each_c_i, each_c_i+each_c_c, 10)
        axes[i].imshow(X_train_aug[random_indices[i],:,:,:],interpolation='nearest')
        axes[i].axis('off')        
    pyplot.show()
0 . Class :  Speed limit (20km/h)
1 . Class :  Speed limit (30km/h)
2 . Class :  Speed limit (50km/h)
3 . Class :  Speed limit (60km/h)
4 . Class :  Speed limit (70km/h)
5 . Class :  Speed limit (80km/h)
6 . Class :  End of speed limit (80km/h)
7 . Class :  Speed limit (100km/h)
8 . Class :  Speed limit (120km/h)
9 . Class :  No passing
10 . Class :  No passing for vehicles over 3.5 metric tons
11 . Class :  Right-of-way at the next intersection
12 . Class :  Priority road
13 . Class :  Yield
14 . Class :  Stop
15 . Class :  No vehicles
16 . Class :  Vehicles over 3.5 metric tons prohibited
17 . Class :  No entry
18 . Class :  General caution
19 . Class :  Dangerous curve to the left
20 . Class :  Dangerous curve to the right
21 . Class :  Double curve
22 . Class :  Bumpy road
23 . Class :  Slippery road
24 . Class :  Road narrows on the right
25 . Class :  Road work
26 . Class :  Traffic signals
27 . Class :  Pedestrians
28 . Class :  Children crossing
29 . Class :  Bicycles crossing
30 . Class :  Beware of ice/snow
31 . Class :  Wild animals crossing
32 . Class :  End of all speed and passing limits
33 . Class :  Turn right ahead
34 . Class :  Turn left ahead
35 . Class :  Ahead only
36 . Class :  Go straight or right
37 . Class :  Go straight or left
38 . Class :  Keep right
39 . Class :  Keep left
40 . Class :  Roundabout mandatory
41 . Class :  End of no passing
42 . Class :  End of no passing by vehicles over 3.5 metric tons

Model Architecture

In [20]:
## Train your model here.
## Calculate and report the accuracy on the training and validation set.
## Once a final model architecture is selected, 
## the accuracy on the test set should be calculated and reported as well.
## Feel free to use as many code cells as needed.


# here i use densenet model, which is more smaller and efficienter. But it waste too many GPU memory and a little slow.
# this is principle : https://github.com/taki0112/Densenet-Tensorflow
# this is code example of densenet : https://github.com/taki0112/Densenet-Tensorflow/blob/master/MNIST/Densenet_MNIST.py

#  i use Mini-batch , MomentumOptimizer ,l2-regulation, weight-decay

def get_weight(shape,name=None):
    initial = tf.truncated_normal(shape, stddev=0.01)
    return tf.Variable(initial)

def get_bias(shape,name=None):
    initial = tf.constant(0.01, shape=shape)
    return tf.Variable(initial)

def batch_norm(input, is_train=None):
    return tf.contrib.layers.batch_norm(input, scale=True, is_training=is_train, updates_collections=None)
    

def conv2d(input, in_layers, out_layers, kernel_size, with_bias=False):
    W = get_weight([ kernel_size, kernel_size, in_layers, out_layers ])
    conv = tf.nn.conv2d(input, W, [ 1, 1, 1, 1 ], padding='SAME')
    if with_bias:
        return conv + get_bias([ out_layers ])
    return conv


def batch_activ_conv(current, in_layers, out_layers, kernel_size, is_training, keep_prob):
    current = tf.contrib.layers.batch_norm(current, scale=True, is_training=is_training, updates_collections=None)
    current = tf.nn.relu(current)
    current = conv2d(current, in_layers, out_layers, kernel_size)
    current = tf.nn.dropout(current, keep_prob)
    return current

def dense_block(input, num_layers, in_layers, growth, is_training, keep_prob):
    output = input
    out_layers = in_layers
    for index in range(num_layers):
        tmp = batch_activ_conv(output, out_layers, growth, 3, is_training, keep_prob)
        output = tf.concat( [output, tmp], 3)
        out_layers += growth
    return output, out_layers


def avg_pool(input, s):
    return tf.nn.avg_pool(input, [ 1, s, s, 1 ], [1, s, s, 1 ], 'VALID')

def full_connect(input,input_layers,out_layers):
    output = tf.reshape(input, [ -1, input_layers ])
    Wfc = get_weight([ input_layers, out_layers ])
    bfc = get_bias([ out_layers ])
    logits = tf.matmul(output, Wfc) + bfc 
    return logits
In [34]:
# define densenet model

def DenseNet(X, y , keep_prob, is_train, depth=40):
    
    num_layers = int((depth-4) / 3)
#     print('number of features : ',num_layers)
    
    #Inital convolution
    conv_1 = conv2d(X, 3, 16, 3)
#     print('input shape: ',X.shape)
#     print('kern size: 3*3','padding size: 1*1')
#     print('conv_1 shape: ',conv_1.shape)
    
    #First DenseBlock
    block_1, features = dense_block(conv_1, num_layers, 16, 12, is_train, keep_prob)
    
    bac_1 = batch_activ_conv(block_1, features, features, 1, is_train, keep_prob)
    pool_1 = avg_pool(bac_1, 2)
    
#     print('block_1 shape: ',block_1.shape)
#     print('features number: ',features)
#     print('kern size: 12*12','padding size: 1*1')
#     print('bac_1 shape: ',bac_1.shape)
#     print('pool_1 shape: ',pool_1.shape)
    
    #Second DenseBlock
    block_2, features = dense_block(pool_1, num_layers, features, 12, is_train, keep_prob)
    
    bac_2 = batch_activ_conv(block_2, features, features, 1, is_train, keep_prob)
    pool_2 = avg_pool(bac_2, 2)
    
#     print('block_2 shape: ',block_2.shape)
#     print('features number: ',features)
#     print('kern size: 12*12','padding size: 1*1')
#     print('bac_2 shape: ',bac_2.shape)
#     print('pool_2 shape: ',pool_2.shape)
    
    #Third DenseBlock
    block_3, features = dense_block(pool_2, num_layers, features, 12, is_train, keep_prob)
    
#     print('block_3 shape: ',block_3.shape)
#     print('features number: ',features)
    
    bn_3 = batch_norm(block_3,is_train)
    relu_3 = tf.nn.relu(bn_3)
    pool_3 = avg_pool(relu_3, 8)
    final_layers = features
#     print('bn_3 shape: ',bn_3.shape)
#     print('relu_3 shape: ',relu_3.shape)
#     print('pool_3 shape: ',pool_3.shape)
#     print('final features: ', final_layers)
    
    logits=full_connect(pool_3,final_layers,43) 
#     print('output shape',logits.shape)
    return logits

Train, Validate and Test the Model

A validation set can be used to assess how well the model is performing. A low accuracy on the training and validation sets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.

DenseNet Model Initalising traffic sign classification model

Inital convolution

number of features : 12

input shape: (?, 32, 32, 3)

conv kern size: 3x3 padding size: 1x 1

conv_1 shape: (?, 32, 32, 16)

First DenseBlock

block_1 shape: (?, 32, 32, 160)

block_1 features number: 160

conv kern size: 12x12 padding size: 1x 1

bac_1 shape: (?, 32, 32, 160)

pool_1 shape: (?, 16, 16, 160) kern size 2*2

Second DenseBlock

block_2 shape: (?, 16, 16, 304)

block_2 features number: 304

conv kern size: 12x12 padding size: 1x 1

bac_2 shape: (?, 16, 16, 304)

pool_2 shape: (?, 8, 8, 304)kern size 8x8

Third DenseBlock

block_3 shape: (?, 8, 8, 448)

features number: 448

bn_3 shape: (?, 8, 8, 448)

relu_3 shape: (?, 8, 8, 448)

pool_3 shape: (?, 1, 1, 448)kern size 8x8

final features: 448

Output

output shape (?, 43)

In [35]:
# Initial parameters

#Dataset parameters
IMAGE_HEIGHT = 32
IMAGE_WIDTH = 32
IMAGE_CHANNELS = 3
NO_CLASSES = 43

# normaly the epochs can be setted to 100, but i think it is too slow, so i just set to 10.
EPOCHS = 30
BATCH_SIZE = 64


LEARNING_RATE =0.01
KEEP_PROB = 0.9
WEIGHT_DECAY=0

# build store folders
def get_dir(directory):
    if not os.path.exists(directory):
        os.makedirs(directory)
    return directory


SAVE_DIR = get_dir('SAVE/')
MODEL_SAVE_DIR = get_dir(SAVE_DIR + 'Models/')
In [36]:
# define all the tensorflow operation in this class, so that we can easily access.
class DenseNetStarter:
    def __init__(self, session, learning_rate=0.1, weight_decay=0.0001, keep_prob = 0.8):

        self.sess = session
        self.learning_rate_init = learning_rate
        self.weight_decay = weight_decay
        self.input_keep_prob = keep_prob
        self.init_graph()

    def start_training(self, batch_X, batch_y):

        feed_dict = {self.input_image : batch_X, self.y : batch_y, self.keep_prob: self.input_keep_prob ,self.is_train : True, self.learning_rate : self.learning_rate_init}

        _, loss, global_step = self.sess.run([self.train_step, self.mean_loss,  self.global_step, ], feed_dict=feed_dict)
        
        # show the variation of accuracy and loss each 200 interation
        if global_step % 200 ==0:
            train_accuracy = self.sess.run(self.accuracy_operation, feed_dict= {self.input_image : batch_X, self.y : batch_y, self.keep_prob :1.0, self.is_train :False})
            print("Iteration ", global_step, ": with training loss = ", loss, " and accuracy of " , train_accuracy)

            
        ##Code to change the learning rate with the number or epochs needed

    def init_graph(self):

        print("Initalising traffic sign classification model")
        #Define the inputs for the model
        with tf.name_scope('Inputs'):
            self.input_image = tf.placeholder(tf.float32, shape=[None, IMAGE_HEIGHT, IMAGE_WIDTH, IMAGE_CHANNELS], name='Input_X')
            self.y = tf.placeholder(tf.int32, (None), name ='Input_y')
             # perform onehot for y-data
            self.one_hot_y = tf.one_hot(self.y, NO_CLASSES)
            self.is_train = tf.placeholder(tf.bool)
            self.keep_prob = tf.placeholder(tf.float32)
            self.learning_rate = tf.placeholder(tf.float32)
            self.k = tf.placeholder(tf.int32)
            

        ##Run batch through model to get logits
        self.y_out = DenseNet(self.input_image, self.one_hot_y, self.keep_prob ,self.is_train)


        #Calculate the losses
        with tf.name_scope('Train'):
            
            self.l2 = tf.add_n([tf.nn.l2_loss(var) for var in tf.trainable_variables()])
            self.total_loss = tf.nn.softmax_cross_entropy_with_logits(labels=self.one_hot_y , logits=self.y_out)
            self.mean_loss = tf.reduce_mean(self.total_loss)
            self.loss = self.mean_loss + self.l2*self.weight_decay
           
            self.correct_prediction = tf.equal(tf.argmax(self.y_out, 1), tf.argmax( self.one_hot_y,1))
            self.accuracy_operation = tf.reduce_mean(tf.cast(self.correct_prediction, tf.float32))
            
            # get model's softmax probabilities to show the certainty of the model's predictions
            self.probs = tf.nn.softmax(self.y_out)
            self.top_probs = tf.nn.top_k(self.probs, self.k)
            
    

        with tf.name_scope('Optimiser'):
            self.global_step = tf.Variable(0, trainable=False, name='global_step')
            self.optimizer = tf.train.MomentumOptimizer(self.learning_rate, 0.9, use_nesterov=True)
            self.train_step = self.optimizer.minimize(self.loss, global_step = self.global_step)



    def evaluate(self, X, y, max_batch = 128):
        num_examples = len(X)
        total_acc = 0 
        total_loss =0
        
        for offset in range(0, num_examples, max_batch):
            batch_X , batch_y = X[offset:offset + max_batch], y[offset: offset + max_batch]
            # keep prob =1.0
            feed_dict = {self.input_image : batch_X, self.y : batch_y, self.is_train : False, self.keep_prob: 1.0}
            accuracy, loss = self.sess.run([self.accuracy_operation, self.mean_loss ], feed_dict= feed_dict)
            total_acc += (accuracy * len(batch_X))
            total_loss += (loss * len(batch_X))
        return total_acc / num_examples, total_loss / num_examples
    
    def classify_images(self, X, top_k =5):
        num_images = len(X)
        feed_dict = {self.input_image : X, self.is_train : False, self.keep_prob: 1.0, self.k : top_k}
        probs = self.sess.run(self.top_probs, feed_dict= feed_dict)
        return probs
In [37]:
# use to store the accuracy and loss of each dataset
valid_hist_acc = []
train_hist_acc = []
valid_hist_loss = []
train_hist_loss = []
tf.reset_default_graph()

with tf.Session() as sess:

    print('DenseNet Model')
    TrafficNet = DenseNetStarter(sess, LEARNING_RATE, WEIGHT_DECAY, KEEP_PROB)

        
    Saver = tf.train.Saver(max_to_keep=100)
    sess.run(tf.global_variables_initializer())
        
    num_examples = len(X_train_aug)
    print("Training...")
    print()
    for i in range(len(valid_hist_acc), EPOCHS):
        X_train_aug, y_train_aug = shuffle(X_train_aug, y_train_aug)
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train_aug[offset:end], y_train_aug[offset:end]
            TrafficNet.start_training(batch_x, batch_y)
            
        validation_accuracy, valid_loss = TrafficNet.evaluate(X_valid, y_valid, max_batch = BATCH_SIZE)
        training_accuracy, training_loss = TrafficNet.evaluate(X_train, y_train, max_batch = BATCH_SIZE)
        valid_hist_acc.append(validation_accuracy)
        train_hist_acc.append(training_accuracy)
        valid_hist_loss.append(valid_loss)
        train_hist_loss.append(training_loss)
        print("EPOCH {} ...".format(i+1))
        print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        print("Training Accuracy = {:.3f}".format(training_accuracy))
        print()
        
#         plot_training(train_hist, valid_hist)
        if i % 1 ==0:
            print('Saving model ...')
            Saver.save(sess, MODEL_SAVE_DIR + 'densenet.ckpt', global_step=i)
            
#         if validation_accuracy>=0.99:
#             break
            
DenseNet Model
Initalising traffic sign classification model
Training...

Iteration  200 : with training loss =  2.543078  and accuracy of  0.015625
Iteration  400 : with training loss =  2.0223236  and accuracy of  0.0
Iteration  600 : with training loss =  1.377737  and accuracy of  0.0625
Iteration  800 : with training loss =  0.7608124  and accuracy of  0.0
Iteration  1000 : with training loss =  0.4602877  and accuracy of  0.015625
Iteration  1200 : with training loss =  0.3219059  and accuracy of  0.109375
Iteration  1400 : with training loss =  0.21016732  and accuracy of  0.390625
Iteration  1600 : with training loss =  0.20420977  and accuracy of  0.4375
Iteration  1800 : with training loss =  0.24702291  and accuracy of  0.28125
Iteration  2000 : with training loss =  0.20524386  and accuracy of  0.234375
Iteration  2200 : with training loss =  0.21965015  and accuracy of  0.546875
Iteration  2400 : with training loss =  0.09839332  and accuracy of  0.71875
Iteration  2600 : with training loss =  0.08158159  and accuracy of  0.375
EPOCH 1 ...
Validation Accuracy = 0.916
Training Accuracy = 0.931

Saving model ...
Iteration  2800 : with training loss =  0.15785906  and accuracy of  0.828125
Iteration  3000 : with training loss =  0.0634043  and accuracy of  0.8125
Iteration  3200 : with training loss =  0.12286081  and accuracy of  0.65625
Iteration  3400 : with training loss =  0.0752698  and accuracy of  0.890625
Iteration  3600 : with training loss =  0.10026997  and accuracy of  0.875
Iteration  3800 : with training loss =  0.17107546  and accuracy of  0.828125
Iteration  4000 : with training loss =  0.038597323  and accuracy of  0.8125
Iteration  4200 : with training loss =  0.06976554  and accuracy of  0.890625
Iteration  4400 : with training loss =  0.033397775  and accuracy of  0.953125
Iteration  4600 : with training loss =  0.08510621  and accuracy of  0.953125
Iteration  4800 : with training loss =  0.09217261  and accuracy of  0.921875
Iteration  5000 : with training loss =  0.038708664  and accuracy of  0.90625
Iteration  5200 : with training loss =  0.043224216  and accuracy of  0.96875
EPOCH 2 ...
Validation Accuracy = 0.886
Training Accuracy = 0.923

Saving model ...
Iteration  5400 : with training loss =  0.030438954  and accuracy of  0.796875
Iteration  5600 : with training loss =  0.046263505  and accuracy of  0.921875
Iteration  5800 : with training loss =  0.0724448  and accuracy of  0.921875
Iteration  6000 : with training loss =  0.018076269  and accuracy of  0.9375
Iteration  6200 : with training loss =  0.024380289  and accuracy of  1.0
Iteration  6400 : with training loss =  0.025277834  and accuracy of  0.921875
Iteration  6600 : with training loss =  0.034231517  and accuracy of  0.890625
Iteration  6800 : with training loss =  0.013654417  and accuracy of  0.984375
Iteration  7000 : with training loss =  0.019926347  and accuracy of  0.90625
Iteration  7200 : with training loss =  0.06654454  and accuracy of  0.96875
Iteration  7400 : with training loss =  0.024360042  and accuracy of  0.984375
Iteration  7600 : with training loss =  0.013863643  and accuracy of  1.0
Iteration  7800 : with training loss =  0.008549217  and accuracy of  1.0
Iteration  8000 : with training loss =  0.008528668  and accuracy of  0.84375
EPOCH 3 ...
Validation Accuracy = 0.967
Training Accuracy = 0.988

Saving model ...
Iteration  8200 : with training loss =  0.16155961  and accuracy of  0.953125
Iteration  8400 : with training loss =  0.0132571785  and accuracy of  0.96875
Iteration  8600 : with training loss =  0.0031729485  and accuracy of  0.96875
Iteration  8800 : with training loss =  0.0069752135  and accuracy of  1.0
Iteration  9000 : with training loss =  0.0034055684  and accuracy of  0.90625
Iteration  9200 : with training loss =  0.0030905153  and accuracy of  1.0
Iteration  9400 : with training loss =  0.013271041  and accuracy of  0.953125
Iteration  9600 : with training loss =  0.014071846  and accuracy of  0.859375
Iteration  9800 : with training loss =  0.04052032  and accuracy of  0.703125
Iteration  10000 : with training loss =  0.045909762  and accuracy of  0.96875
Iteration  10200 : with training loss =  0.10600853  and accuracy of  0.953125
Iteration  10400 : with training loss =  0.0040907916  and accuracy of  0.90625
Iteration  10600 : with training loss =  0.039872095  and accuracy of  0.984375
EPOCH 4 ...
Validation Accuracy = 0.978
Training Accuracy = 0.997

Saving model ...
Iteration  10800 : with training loss =  0.057998817  and accuracy of  0.984375
Iteration  11000 : with training loss =  0.003260717  and accuracy of  1.0
Iteration  11200 : with training loss =  0.00326199  and accuracy of  1.0
Iteration  11400 : with training loss =  0.012099727  and accuracy of  0.984375
Iteration  11600 : with training loss =  0.0168255  and accuracy of  0.96875
Iteration  11800 : with training loss =  0.012309151  and accuracy of  0.96875
Iteration  12000 : with training loss =  0.0075980537  and accuracy of  1.0
Iteration  12200 : with training loss =  0.08032811  and accuracy of  0.984375
Iteration  12400 : with training loss =  0.0060910517  and accuracy of  0.953125
Iteration  12600 : with training loss =  0.008526121  and accuracy of  0.984375
Iteration  12800 : with training loss =  0.005148223  and accuracy of  1.0
Iteration  13000 : with training loss =  0.00766856  and accuracy of  1.0
Iteration  13200 : with training loss =  0.017267283  and accuracy of  1.0
Iteration  13400 : with training loss =  0.0044879513  and accuracy of  1.0
EPOCH 5 ...
Validation Accuracy = 0.980
Training Accuracy = 0.991

Saving model ...
Iteration  13600 : with training loss =  0.00640042  and accuracy of  0.953125
Iteration  13800 : with training loss =  0.028952215  and accuracy of  0.984375
Iteration  14000 : with training loss =  0.01694232  and accuracy of  0.96875
Iteration  14200 : with training loss =  0.005172129  and accuracy of  0.984375
Iteration  14400 : with training loss =  0.0019588333  and accuracy of  1.0
Iteration  14600 : with training loss =  0.051200647  and accuracy of  0.953125
Iteration  14800 : with training loss =  0.0021954058  and accuracy of  0.96875
Iteration  15000 : with training loss =  0.07619715  and accuracy of  0.984375
Iteration  15200 : with training loss =  0.001122107  and accuracy of  0.984375
Iteration  15400 : with training loss =  0.0023864708  and accuracy of  1.0
Iteration  15600 : with training loss =  0.0013865073  and accuracy of  0.984375
Iteration  15800 : with training loss =  0.022235168  and accuracy of  0.984375
Iteration  16000 : with training loss =  0.027952407  and accuracy of  0.984375
EPOCH 6 ...
Validation Accuracy = 0.967
Training Accuracy = 0.982

Saving model ...
Iteration  16200 : with training loss =  0.022412898  and accuracy of  1.0
Iteration  16400 : with training loss =  0.026953291  and accuracy of  0.984375
Iteration  16600 : with training loss =  0.0012689147  and accuracy of  0.984375
Iteration  16800 : with training loss =  0.004021172  and accuracy of  0.984375
Iteration  17000 : with training loss =  0.008900538  and accuracy of  0.96875
Iteration  17200 : with training loss =  0.0027390164  and accuracy of  1.0
Iteration  17400 : with training loss =  0.0018729226  and accuracy of  0.96875
Iteration  17600 : with training loss =  0.002230503  and accuracy of  0.921875
Iteration  17800 : with training loss =  0.054173075  and accuracy of  0.96875
Iteration  18000 : with training loss =  0.03352803  and accuracy of  0.96875
Iteration  18200 : with training loss =  0.0086866645  and accuracy of  0.984375
Iteration  18400 : with training loss =  0.0026851052  and accuracy of  0.984375
Iteration  18600 : with training loss =  0.0036634204  and accuracy of  0.984375
Iteration  18800 : with training loss =  0.011001081  and accuracy of  0.96875
EPOCH 7 ...
Validation Accuracy = 0.966
Training Accuracy = 0.990

Saving model ...
Iteration  19000 : with training loss =  0.060921915  and accuracy of  1.0
Iteration  19200 : with training loss =  0.020752456  and accuracy of  0.984375
Iteration  19400 : with training loss =  0.0027591996  and accuracy of  1.0
Iteration  19600 : with training loss =  0.0016875282  and accuracy of  1.0
Iteration  19800 : with training loss =  0.0022801557  and accuracy of  1.0
Iteration  20000 : with training loss =  0.003641712  and accuracy of  1.0
Iteration  20200 : with training loss =  0.004635774  and accuracy of  0.984375
Iteration  20400 : with training loss =  0.004793978  and accuracy of  0.984375
Iteration  20600 : with training loss =  0.0073654125  and accuracy of  0.96875
Iteration  20800 : with training loss =  0.004467535  and accuracy of  1.0
Iteration  21000 : with training loss =  0.0050022397  and accuracy of  0.984375
Iteration  21200 : with training loss =  0.0055701425  and accuracy of  0.9375
Iteration  21400 : with training loss =  0.0009222034  and accuracy of  1.0
EPOCH 8 ...
Validation Accuracy = 0.984
Training Accuracy = 0.998

Saving model ...
Iteration  21600 : with training loss =  0.0043108207  and accuracy of  1.0
Iteration  21800 : with training loss =  0.0017633055  and accuracy of  0.984375
Iteration  22000 : with training loss =  0.0011309612  and accuracy of  1.0
Iteration  22200 : with training loss =  0.0043325177  and accuracy of  0.984375
Iteration  22400 : with training loss =  0.010827651  and accuracy of  0.96875
Iteration  22600 : with training loss =  0.0018001888  and accuracy of  0.984375
Iteration  22800 : with training loss =  0.008079341  and accuracy of  0.953125
Iteration  23000 : with training loss =  0.002692148  and accuracy of  1.0
Iteration  23200 : with training loss =  0.014333847  and accuracy of  0.96875
Iteration  23400 : with training loss =  0.015186948  and accuracy of  1.0
Iteration  23600 : with training loss =  0.0063665677  and accuracy of  1.0
Iteration  23800 : with training loss =  0.0018158867  and accuracy of  1.0
Iteration  24000 : with training loss =  0.0007431196  and accuracy of  1.0
EPOCH 9 ...
Validation Accuracy = 0.968
Training Accuracy = 0.989

Saving model ...
Iteration  24200 : with training loss =  0.0015154125  and accuracy of  1.0
Iteration  24400 : with training loss =  0.0008332193  and accuracy of  1.0
Iteration  24600 : with training loss =  0.0031202286  and accuracy of  1.0
Iteration  24800 : with training loss =  0.00051550096  and accuracy of  0.984375
Iteration  25000 : with training loss =  0.001989665  and accuracy of  0.984375
Iteration  25200 : with training loss =  0.036375802  and accuracy of  1.0
Iteration  25400 : with training loss =  0.007154731  and accuracy of  0.984375
Iteration  25600 : with training loss =  0.001224816  and accuracy of  1.0
Iteration  25800 : with training loss =  0.001736267  and accuracy of  1.0
Iteration  26000 : with training loss =  0.00047391848  and accuracy of  1.0
Iteration  26200 : with training loss =  0.0015473432  and accuracy of  1.0
Iteration  26400 : with training loss =  0.009439414  and accuracy of  1.0
Iteration  26600 : with training loss =  0.0018676791  and accuracy of  1.0
Iteration  26800 : with training loss =  0.0017509675  and accuracy of  1.0
EPOCH 10 ...
Validation Accuracy = 0.982
Training Accuracy = 0.998

Saving model ...
Iteration  27000 : with training loss =  0.0013098232  and accuracy of  1.0
Iteration  27200 : with training loss =  0.0024172165  and accuracy of  1.0
Iteration  27400 : with training loss =  0.0067001204  and accuracy of  0.984375
Iteration  27600 : with training loss =  0.018320492  and accuracy of  1.0
Iteration  27800 : with training loss =  0.047384024  and accuracy of  0.984375
Iteration  28000 : with training loss =  0.00253492  and accuracy of  0.984375
Iteration  28200 : with training loss =  0.004608659  and accuracy of  1.0
Iteration  28400 : with training loss =  0.00044758787  and accuracy of  1.0
Iteration  28600 : with training loss =  0.00088201824  and accuracy of  1.0
Iteration  28800 : with training loss =  0.0004541147  and accuracy of  1.0
Iteration  29000 : with training loss =  0.00050909305  and accuracy of  1.0
Iteration  29200 : with training loss =  0.00133823  and accuracy of  0.984375
Iteration  29400 : with training loss =  0.00057172973  and accuracy of  1.0
EPOCH 11 ...
Validation Accuracy = 0.981
Training Accuracy = 0.998

Saving model ...
Iteration  29600 : with training loss =  0.00086258736  and accuracy of  1.0
Iteration  29800 : with training loss =  0.0026472723  and accuracy of  0.984375
Iteration  30000 : with training loss =  0.0012911698  and accuracy of  0.96875
Iteration  30200 : with training loss =  0.001951825  and accuracy of  1.0
Iteration  30400 : with training loss =  0.00059756625  and accuracy of  1.0
Iteration  30600 : with training loss =  0.0018284972  and accuracy of  1.0
Iteration  30800 : with training loss =  0.004657857  and accuracy of  1.0
Iteration  31000 : with training loss =  0.0004685403  and accuracy of  1.0
Iteration  31200 : with training loss =  0.0016792663  and accuracy of  1.0
Iteration  31400 : with training loss =  0.0044574  and accuracy of  1.0
Iteration  31600 : with training loss =  0.008713706  and accuracy of  1.0
Iteration  31800 : with training loss =  0.00040348026  and accuracy of  0.984375
Iteration  32000 : with training loss =  0.0018237086  and accuracy of  1.0
Iteration  32200 : with training loss =  0.0058223857  and accuracy of  1.0
EPOCH 12 ...
Validation Accuracy = 0.988
Training Accuracy = 0.999

Saving model ...
Iteration  32400 : with training loss =  0.00048849045  and accuracy of  1.0
Iteration  32600 : with training loss =  0.0075978814  and accuracy of  1.0
Iteration  32800 : with training loss =  0.0010621343  and accuracy of  1.0
Iteration  33000 : with training loss =  0.0008159641  and accuracy of  1.0
Iteration  33200 : with training loss =  0.011001874  and accuracy of  1.0
Iteration  33400 : with training loss =  0.0005623444  and accuracy of  1.0
Iteration  33600 : with training loss =  0.0029805948  and accuracy of  0.96875
Iteration  33800 : with training loss =  0.025923602  and accuracy of  0.96875
Iteration  34000 : with training loss =  0.015175416  and accuracy of  0.984375
Iteration  34200 : with training loss =  0.0015827549  and accuracy of  1.0
Iteration  34400 : with training loss =  0.0035901803  and accuracy of  0.984375
Iteration  34600 : with training loss =  0.0030657686  and accuracy of  1.0
Iteration  34800 : with training loss =  0.0001939122  and accuracy of  1.0
EPOCH 13 ...
Validation Accuracy = 0.987
Training Accuracy = 1.000

Saving model ...
Iteration  35000 : with training loss =  0.0019064945  and accuracy of  1.0
Iteration  35200 : with training loss =  0.0011699339  and accuracy of  1.0
Iteration  35400 : with training loss =  0.0032901461  and accuracy of  1.0
Iteration  35600 : with training loss =  0.0006879144  and accuracy of  1.0
Iteration  35800 : with training loss =  0.0004960821  and accuracy of  1.0
Iteration  36000 : with training loss =  0.010354874  and accuracy of  1.0
Iteration  36200 : with training loss =  0.010359341  and accuracy of  1.0
Iteration  36400 : with training loss =  0.07699663  and accuracy of  0.984375
Iteration  36600 : with training loss =  0.0008054774  and accuracy of  0.984375
Iteration  36800 : with training loss =  0.022537254  and accuracy of  1.0
Iteration  37000 : with training loss =  0.004503684  and accuracy of  1.0
Iteration  37200 : with training loss =  0.0003312684  and accuracy of  1.0
Iteration  37400 : with training loss =  0.0031484275  and accuracy of  1.0
Iteration  37600 : with training loss =  9.376447e-05  and accuracy of  1.0
EPOCH 14 ...
Validation Accuracy = 0.976
Training Accuracy = 0.994

Saving model ...
Iteration  37800 : with training loss =  0.00029343867  and accuracy of  1.0
Iteration  38000 : with training loss =  0.0026150576  and accuracy of  1.0
Iteration  38200 : with training loss =  0.00035026748  and accuracy of  1.0
Iteration  38400 : with training loss =  0.0016789514  and accuracy of  1.0
Iteration  38600 : with training loss =  0.010512076  and accuracy of  1.0
Iteration  38800 : with training loss =  0.0011450602  and accuracy of  1.0
Iteration  39000 : with training loss =  0.0008485231  and accuracy of  1.0
Iteration  39200 : with training loss =  0.0067219883  and accuracy of  0.984375
Iteration  39400 : with training loss =  0.0017582178  and accuracy of  1.0
Iteration  39600 : with training loss =  0.002571216  and accuracy of  1.0
Iteration  39800 : with training loss =  0.003244803  and accuracy of  1.0
Iteration  40000 : with training loss =  0.006793803  and accuracy of  1.0
Iteration  40200 : with training loss =  0.037584577  and accuracy of  1.0
EPOCH 15 ...
Validation Accuracy = 0.992
Training Accuracy = 0.999

Saving model ...
Iteration  40400 : with training loss =  0.0005362739  and accuracy of  1.0
Iteration  40600 : with training loss =  0.00091973844  and accuracy of  0.984375
Iteration  40800 : with training loss =  0.000624933  and accuracy of  1.0
Iteration  41000 : with training loss =  0.00078879506  and accuracy of  1.0
Iteration  41200 : with training loss =  0.001104421  and accuracy of  1.0
Iteration  41400 : with training loss =  0.0035318492  and accuracy of  1.0
Iteration  41600 : with training loss =  0.0033529317  and accuracy of  1.0
Iteration  41800 : with training loss =  0.00841709  and accuracy of  0.984375
Iteration  42000 : with training loss =  0.00097170187  and accuracy of  0.9375
Iteration  42200 : with training loss =  0.00058363413  and accuracy of  1.0
Iteration  42400 : with training loss =  0.0012190131  and accuracy of  1.0
Iteration  42600 : with training loss =  0.004609032  and accuracy of  1.0
Iteration  42800 : with training loss =  0.0010118161  and accuracy of  1.0
Iteration  43000 : with training loss =  0.001388867  and accuracy of  0.984375
EPOCH 16 ...
Validation Accuracy = 0.983
Training Accuracy = 0.999

Saving model ...
Iteration  43200 : with training loss =  0.00070153276  and accuracy of  1.0
Iteration  43400 : with training loss =  0.01825933  and accuracy of  1.0
Iteration  43600 : with training loss =  0.0014010769  and accuracy of  1.0
Iteration  43800 : with training loss =  0.00082351244  and accuracy of  1.0
Iteration  44000 : with training loss =  0.0034623404  and accuracy of  0.984375
Iteration  44200 : with training loss =  0.00032538024  and accuracy of  1.0
Iteration  44400 : with training loss =  0.00030509965  and accuracy of  1.0
Iteration  44600 : with training loss =  0.0025875783  and accuracy of  1.0
Iteration  44800 : with training loss =  0.0015110754  and accuracy of  1.0
Iteration  45000 : with training loss =  0.00051261217  and accuracy of  1.0
Iteration  45200 : with training loss =  0.0010596915  and accuracy of  1.0
Iteration  45400 : with training loss =  0.00016250127  and accuracy of  1.0
Iteration  45600 : with training loss =  7.875158e-05  and accuracy of  1.0
EPOCH 17 ...
Validation Accuracy = 0.983
Training Accuracy = 0.998

Saving model ...
Iteration  45800 : with training loss =  0.00018703437  and accuracy of  1.0
Iteration  46000 : with training loss =  0.0001123739  and accuracy of  1.0
Iteration  46200 : with training loss =  0.03904605  and accuracy of  1.0
Iteration  46400 : with training loss =  0.0004279383  and accuracy of  1.0
Iteration  46600 : with training loss =  0.016676053  and accuracy of  1.0
Iteration  46800 : with training loss =  0.00094911986  and accuracy of  1.0
Iteration  47000 : with training loss =  0.0019386068  and accuracy of  0.984375
Iteration  47200 : with training loss =  0.0076165125  and accuracy of  1.0
Iteration  47400 : with training loss =  0.0022340983  and accuracy of  1.0
Iteration  47600 : with training loss =  0.001091134  and accuracy of  1.0
Iteration  47800 : with training loss =  0.00045963284  and accuracy of  1.0
Iteration  48000 : with training loss =  0.0054244106  and accuracy of  1.0
Iteration  48200 : with training loss =  0.0016002415  and accuracy of  1.0
EPOCH 18 ...
Validation Accuracy = 0.984
Training Accuracy = 0.998

Saving model ...
Iteration  48400 : with training loss =  0.004639559  and accuracy of  1.0
Iteration  48600 : with training loss =  0.00011777295  and accuracy of  1.0
Iteration  48800 : with training loss =  0.00065147446  and accuracy of  1.0
Iteration  49000 : with training loss =  0.0006680272  and accuracy of  1.0
Iteration  49200 : with training loss =  0.0068023806  and accuracy of  1.0
Iteration  49400 : with training loss =  8.371737e-05  and accuracy of  1.0
Iteration  49600 : with training loss =  0.00065303675  and accuracy of  1.0
Iteration  49800 : with training loss =  0.0046278355  and accuracy of  0.984375
Iteration  50000 : with training loss =  0.0018975323  and accuracy of  1.0
Iteration  50200 : with training loss =  0.0012509609  and accuracy of  1.0
Iteration  50400 : with training loss =  0.020576539  and accuracy of  0.984375
Iteration  50600 : with training loss =  0.001057356  and accuracy of  1.0
Iteration  50800 : with training loss =  0.00094922364  and accuracy of  1.0
Iteration  51000 : with training loss =  0.0019623963  and accuracy of  1.0
EPOCH 19 ...
Validation Accuracy = 0.985
Training Accuracy = 0.999

Saving model ...
Iteration  51200 : with training loss =  0.0019452126  and accuracy of  1.0
Iteration  51400 : with training loss =  0.00084096094  and accuracy of  0.984375
Iteration  51600 : with training loss =  0.00020857721  and accuracy of  1.0
Iteration  51800 : with training loss =  0.00023463523  and accuracy of  1.0
Iteration  52000 : with training loss =  0.014651164  and accuracy of  1.0
Iteration  52200 : with training loss =  0.0011406405  and accuracy of  1.0
Iteration  52400 : with training loss =  0.000251028  and accuracy of  1.0
Iteration  52600 : with training loss =  0.002689634  and accuracy of  1.0
Iteration  52800 : with training loss =  0.0002575221  and accuracy of  1.0
Iteration  53000 : with training loss =  0.001668551  and accuracy of  0.984375
Iteration  53200 : with training loss =  0.05551936  and accuracy of  1.0
Iteration  53400 : with training loss =  0.00016152725  and accuracy of  1.0
Iteration  53600 : with training loss =  0.0019518617  and accuracy of  1.0
EPOCH 20 ...
Validation Accuracy = 0.983
Training Accuracy = 1.000

Saving model ...
Iteration  53800 : with training loss =  0.0017246722  and accuracy of  1.0
Iteration  54000 : with training loss =  0.000516945  and accuracy of  1.0
Iteration  54200 : with training loss =  0.00021416339  and accuracy of  1.0
Iteration  54400 : with training loss =  0.00047588098  and accuracy of  1.0
Iteration  54600 : with training loss =  0.000743094  and accuracy of  1.0
Iteration  54800 : with training loss =  0.006736383  and accuracy of  0.984375
Iteration  55000 : with training loss =  0.0014347297  and accuracy of  1.0
Iteration  55200 : with training loss =  0.00094387564  and accuracy of  1.0
Iteration  55400 : with training loss =  0.0009931523  and accuracy of  1.0
Iteration  55600 : with training loss =  0.0024255235  and accuracy of  1.0
Iteration  55800 : with training loss =  0.0018404436  and accuracy of  1.0
Iteration  56000 : with training loss =  0.00024190579  and accuracy of  1.0
Iteration  56200 : with training loss =  0.0018696658  and accuracy of  1.0
Iteration  56400 : with training loss =  0.0016202363  and accuracy of  1.0
EPOCH 21 ...
Validation Accuracy = 0.986
Training Accuracy = 1.000

Saving model ...
Iteration  56600 : with training loss =  0.013548495  and accuracy of  1.0
Iteration  56800 : with training loss =  0.0013081231  and accuracy of  1.0
Iteration  57000 : with training loss =  0.0001455436  and accuracy of  1.0
Iteration  57200 : with training loss =  0.00011242974  and accuracy of  1.0
Iteration  57400 : with training loss =  0.00339869  and accuracy of  1.0
Iteration  57600 : with training loss =  0.0012075347  and accuracy of  1.0
Iteration  57800 : with training loss =  0.005089052  and accuracy of  1.0
Iteration  58000 : with training loss =  0.00039850417  and accuracy of  1.0
Iteration  58200 : with training loss =  0.0036981767  and accuracy of  1.0
Iteration  58400 : with training loss =  0.0017184209  and accuracy of  1.0
Iteration  58600 : with training loss =  8.252113e-05  and accuracy of  1.0
Iteration  58800 : with training loss =  0.00012427845  and accuracy of  1.0
Iteration  59000 : with training loss =  0.00014539123  and accuracy of  1.0
EPOCH 22 ...
Validation Accuracy = 0.990
Training Accuracy = 0.997

Saving model ...
Iteration  59200 : with training loss =  0.00016869983  and accuracy of  1.0
Iteration  59400 : with training loss =  0.03148279  and accuracy of  1.0
Iteration  59600 : with training loss =  0.0016248131  and accuracy of  0.984375
Iteration  59800 : with training loss =  0.0021778257  and accuracy of  1.0
Iteration  60000 : with training loss =  0.00037424406  and accuracy of  1.0
Iteration  60200 : with training loss =  0.0002862193  and accuracy of  1.0
Iteration  60400 : with training loss =  0.00041149458  and accuracy of  1.0
Iteration  60600 : with training loss =  0.00018251975  and accuracy of  1.0
Iteration  60800 : with training loss =  0.00015556361  and accuracy of  1.0
Iteration  61000 : with training loss =  0.0001890326  and accuracy of  1.0
Iteration  61200 : with training loss =  0.0020967207  and accuracy of  1.0
Iteration  61400 : with training loss =  0.0039883642  and accuracy of  0.984375
Iteration  61600 : with training loss =  0.0002087419  and accuracy of  1.0
Iteration  61800 : with training loss =  0.02808777  and accuracy of  1.0
EPOCH 23 ...
Validation Accuracy = 0.984
Training Accuracy = 0.999

Saving model ...
Iteration  62000 : with training loss =  0.0006679963  and accuracy of  1.0
Iteration  62200 : with training loss =  0.0007805344  and accuracy of  1.0
Iteration  62400 : with training loss =  0.003611296  and accuracy of  1.0
Iteration  62600 : with training loss =  0.0020308187  and accuracy of  1.0
Iteration  62800 : with training loss =  0.003216451  and accuracy of  1.0
Iteration  63000 : with training loss =  0.0002501214  and accuracy of  1.0
Iteration  63200 : with training loss =  0.00020643926  and accuracy of  1.0
Iteration  63400 : with training loss =  0.00041376043  and accuracy of  1.0
Iteration  63600 : with training loss =  0.0012842948  and accuracy of  0.984375
Iteration  63800 : with training loss =  0.0026231348  and accuracy of  0.984375
Iteration  64000 : with training loss =  0.00041757536  and accuracy of  0.984375
Iteration  64200 : with training loss =  8.4481544e-05  and accuracy of  1.0
Iteration  64400 : with training loss =  0.0002920832  and accuracy of  1.0
EPOCH 24 ...
Validation Accuracy = 0.992
Training Accuracy = 1.000

Saving model ...
Iteration  64600 : with training loss =  0.00060471246  and accuracy of  1.0
Iteration  64800 : with training loss =  3.8229344e-05  and accuracy of  1.0
Iteration  65000 : with training loss =  0.00030509743  and accuracy of  1.0
Iteration  65200 : with training loss =  0.00022681401  and accuracy of  1.0
Iteration  65400 : with training loss =  0.00051347725  and accuracy of  1.0
Iteration  65600 : with training loss =  0.00022793746  and accuracy of  1.0
Iteration  65800 : with training loss =  0.0003278777  and accuracy of  1.0
Iteration  66000 : with training loss =  0.0002768166  and accuracy of  1.0
Iteration  66200 : with training loss =  0.00065633934  and accuracy of  1.0
Iteration  66400 : with training loss =  0.00034385978  and accuracy of  1.0
Iteration  66600 : with training loss =  0.00018695174  and accuracy of  1.0
Iteration  66800 : with training loss =  0.00016683186  and accuracy of  1.0
Iteration  67000 : with training loss =  0.0001961732  and accuracy of  1.0
Iteration  67200 : with training loss =  6.387332e-05  and accuracy of  1.0
EPOCH 25 ...
Validation Accuracy = 0.988
Training Accuracy = 1.000

Saving model ...
Iteration  67400 : with training loss =  0.009775224  and accuracy of  1.0
Iteration  67600 : with training loss =  0.0001771622  and accuracy of  1.0
Iteration  67800 : with training loss =  0.00064325315  and accuracy of  1.0
Iteration  68000 : with training loss =  0.00040474473  and accuracy of  1.0
Iteration  68200 : with training loss =  0.0012169555  and accuracy of  0.984375
Iteration  68400 : with training loss =  0.000988933  and accuracy of  1.0
Iteration  68600 : with training loss =  0.0010815914  and accuracy of  1.0
Iteration  68800 : with training loss =  0.0004581197  and accuracy of  1.0
Iteration  69000 : with training loss =  0.00030632823  and accuracy of  1.0
Iteration  69200 : with training loss =  0.00012451429  and accuracy of  1.0
Iteration  69400 : with training loss =  0.0008357784  and accuracy of  1.0
Iteration  69600 : with training loss =  0.00021806255  and accuracy of  1.0
Iteration  69800 : with training loss =  0.021111026  and accuracy of  1.0
EPOCH 26 ...
Validation Accuracy = 0.985
Training Accuracy = 1.000

Saving model ...
Iteration  70000 : with training loss =  0.0004838014  and accuracy of  1.0
Iteration  70200 : with training loss =  0.0002741923  and accuracy of  1.0
Iteration  70400 : with training loss =  0.00010999689  and accuracy of  1.0
Iteration  70600 : with training loss =  0.00030761544  and accuracy of  1.0
Iteration  70800 : with training loss =  9.7516844e-05  and accuracy of  1.0
Iteration  71000 : with training loss =  0.0006182977  and accuracy of  1.0
Iteration  71200 : with training loss =  0.00023319316  and accuracy of  1.0
Iteration  71400 : with training loss =  0.0004410864  and accuracy of  1.0
Iteration  71600 : with training loss =  0.002393341  and accuracy of  1.0
Iteration  71800 : with training loss =  0.0001269193  and accuracy of  1.0
Iteration  72000 : with training loss =  0.00019469169  and accuracy of  1.0
Iteration  72200 : with training loss =  0.002452534  and accuracy of  1.0
Iteration  72400 : with training loss =  0.010042537  and accuracy of  0.984375
EPOCH 27 ...
Validation Accuracy = 0.993
Training Accuracy = 1.000

Saving model ...
Iteration  72600 : with training loss =  0.0011084855  and accuracy of  1.0
Iteration  72800 : with training loss =  0.001351058  and accuracy of  1.0
Iteration  73000 : with training loss =  0.00031011374  and accuracy of  1.0
Iteration  73200 : with training loss =  0.0003390242  and accuracy of  1.0
Iteration  73400 : with training loss =  0.0042299936  and accuracy of  0.984375
Iteration  73600 : with training loss =  0.00038553707  and accuracy of  1.0
Iteration  73800 : with training loss =  0.00077684544  and accuracy of  1.0
Iteration  74000 : with training loss =  0.0017400312  and accuracy of  1.0
Iteration  74200 : with training loss =  0.000520559  and accuracy of  1.0
Iteration  74400 : with training loss =  0.00048338875  and accuracy of  1.0
Iteration  74600 : with training loss =  0.0006363201  and accuracy of  1.0
Iteration  74800 : with training loss =  0.003300791  and accuracy of  1.0
Iteration  75000 : with training loss =  0.0018887701  and accuracy of  1.0
Iteration  75200 : with training loss =  6.0692124e-05  and accuracy of  1.0
EPOCH 28 ...
Validation Accuracy = 0.987
Training Accuracy = 1.000

Saving model ...
Iteration  75400 : with training loss =  9.251529e-05  and accuracy of  1.0
Iteration  75600 : with training loss =  0.00019177153  and accuracy of  1.0
Iteration  75800 : with training loss =  0.00036230654  and accuracy of  1.0
Iteration  76000 : with training loss =  0.00018042207  and accuracy of  1.0
Iteration  76200 : with training loss =  0.00024339792  and accuracy of  1.0
Iteration  76400 : with training loss =  5.0244988e-05  and accuracy of  1.0
Iteration  76600 : with training loss =  0.0005864715  and accuracy of  1.0
Iteration  76800 : with training loss =  0.00010892954  and accuracy of  1.0
Iteration  77000 : with training loss =  0.0007489055  and accuracy of  1.0
Iteration  77200 : with training loss =  0.00028073613  and accuracy of  1.0
Iteration  77400 : with training loss =  0.004002605  and accuracy of  1.0
Iteration  77600 : with training loss =  0.0021050652  and accuracy of  1.0
Iteration  77800 : with training loss =  0.00029840713  and accuracy of  1.0
EPOCH 29 ...
Validation Accuracy = 0.985
Training Accuracy = 0.999

Saving model ...
Iteration  78000 : with training loss =  0.0069352337  and accuracy of  1.0
Iteration  78200 : with training loss =  0.0003218985  and accuracy of  1.0
Iteration  78400 : with training loss =  0.00026323224  and accuracy of  1.0
Iteration  78600 : with training loss =  0.00014565137  and accuracy of  1.0
Iteration  78800 : with training loss =  4.8010203e-05  and accuracy of  1.0
Iteration  79000 : with training loss =  0.0002319814  and accuracy of  1.0
Iteration  79200 : with training loss =  0.0004946013  and accuracy of  1.0
Iteration  79400 : with training loss =  0.00022410814  and accuracy of  1.0
Iteration  79600 : with training loss =  1.949724e-05  and accuracy of  1.0
Iteration  79800 : with training loss =  0.00021236732  and accuracy of  1.0
Iteration  80000 : with training loss =  0.00013404764  and accuracy of  1.0
Iteration  80200 : with training loss =  0.00057925674  and accuracy of  1.0
Iteration  80400 : with training loss =  0.010595691  and accuracy of  1.0
Iteration  80600 : with training loss =  0.00046735117  and accuracy of  1.0
EPOCH 30 ...
Validation Accuracy = 0.988
Training Accuracy = 1.000

Saving model ...
In [38]:
# plot the diagram of variation of accuracy and loss
def plot_training(train_hist_acc_in, valid_hist_acc_in, train_hist_loss_in, valid_hist_loss_in):
    curves_figure = pyplot.figure(figsize = (10, 4))
    axis = curves_figure.add_subplot(1,2,1)
    axis.plot(train_hist_acc_in, label='train')
    axis.plot(valid_hist_acc_in, label='valid')
    pyplot.grid()
    pyplot.legend()
    pyplot.xlabel("epoch")
    pyplot.ylabel("accuracy")
    
    axis = curves_figure.add_subplot(1,2,2)
    axis.plot(train_hist_loss_in, label='train')
    axis.plot(valid_hist_loss_in, label='valid')
    pyplot.grid()
    pyplot.legend()
    pyplot.xlabel("epoch")
    pyplot.ylabel("cross-entropy")



plot_training( train_hist_acc, valid_hist_acc,  train_hist_loss, valid_hist_loss )
In [40]:
# restore the model and test it in test_dataset
# test_data need to be preprocessed

SAVE_DIR = get_dir('SAVE/')
MODEL_SAVE_DIR = get_dir(SAVE_DIR + 'Models/')
MODEL_LOAD_PATH = './' + MODEL_SAVE_DIR + 'densenet.ckpt-29'

tf.reset_default_graph()

with tf.Session() as sess:
    TrafficNet = DenseNetStarter(sess, LEARNING_RATE, WEIGHT_DECAY, KEEP_PROB)
    Saver = tf.train.Saver(max_to_keep=100)
    sess.run(tf.global_variables_initializer())
    if MODEL_LOAD_PATH is not None:
        Saver.restore(sess, MODEL_LOAD_PATH)
        print('Model restored from ' + MODEL_LOAD_PATH)
        acc, loss = TrafficNet.evaluate(X_test, y_test, max_batch = BATCH_SIZE)
        
        print("Final DenseNet accuracy on the test set ", acc*100, "%")
tf.reset_default_graph()
Initalising traffic sign classification model
INFO:tensorflow:Restoring parameters from ./SAVE/Models/densenet.ckpt-29
Model restored from ./SAVE/Models/densenet.ckpt-29
Final DenseNet accuracy on the test set  98.22644497417592 %

Step 3: Test a Model on New Images

To give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Load and Output the Images

In [42]:
### Load the images and plot them here.
### Feel free to use as many code cells as needed.
### Load the images and plot them here.
### Feel free to use as many code cells as needed.

def load_images(folder):
    images = []
    for filename in os.listdir(folder):
        img = cv2.imread(os.path.join(folder,filename))
        if img is not None:
            images.append(img)
    return images

images = load_images('./downloaded_signs')

# Resize images and stack them
for i, image in enumerate(images):
    temp = np.copy(image)
    temp = cv2.cvtColor(temp, cv2.COLOR_BGR2RGB)
    images[i] = np.expand_dims(cv2.resize(temp, (32, 32)), axis=0)
    
figs,axes=plt.subplots(1,9,figsize=(9,1))
figs.subplots_adjust(left = 0, right = 1, bottom = 0, top = 1, hspace = 0.05, wspace = 0.05)
  
im_data = np.vstack(images)
for i, image in enumerate(im_data):

    axes[i].imshow(image)
    axes[i].axis('off')
    
pyplot.show()

Predict the Sign Type for Each Image

In [43]:
### Run the predictions here and use the model to output the prediction for each image.
### Make sure to pre-process the images with the same pre-processing pipeline used earlier.
### Feel free to use as many code cells as needed.
# load sign_names
sign_names = read_csv("signnames.csv").values[:, 1]
# manual label these images
y_data = np.array([7,0,1,5,18,17,25,36,13])

# preprocess these images
im_data_norm = normalisation(im_data)

SAVE_DIR = get_dir('SAVE/')
MODEL_SAVE_DIR = get_dir(SAVE_DIR + 'Models/')
MODEL_LOAD_PATH = './' + MODEL_SAVE_DIR + 'densenet.ckpt-29'

tf.reset_default_graph()

with tf.Session() as sess:
    
    TrafficNet = DenseNetStarter(sess, LEARNING_RATE, WEIGHT_DECAY, KEEP_PROB)
    Saver = tf.train.Saver(max_to_keep=100)
    sess.run(tf.global_variables_initializer())
    if MODEL_LOAD_PATH is not None:
        Saver.restore(sess, MODEL_LOAD_PATH)
        print('Model restored from ' + MODEL_LOAD_PATH)
        values_dense, indices_dense = TrafficNet.classify_images(im_data_norm)
tf.reset_default_graph()

# compare results
for i in range(len(y_data)):
    plt.figure()
    plt.imshow(im_data_norm[i])
    plt.show()
    print('Correct label: ' + sign_names[y_data[i]] )
    print('Prediced label: ' + sign_names[indices_dense[i]][0] )
    print('Match with the Ground Truth : ', str(values_dense[i][0]))
    print()
Initalising traffic sign classification model
INFO:tensorflow:Restoring parameters from ./SAVE/Models/densenet.ckpt-29
Model restored from ./SAVE/Models/densenet.ckpt-29
Correct label: Speed limit (100km/h)
Prediced label: Speed limit (100km/h)
Match with the Ground Truth :  0.99990165

Correct label: Speed limit (20km/h)
Prediced label: Speed limit (20km/h)
Match with the Ground Truth :  0.9996489

Correct label: Speed limit (30km/h)
Prediced label: Speed limit (30km/h)
Match with the Ground Truth :  0.9999839

Correct label: Speed limit (80km/h)
Prediced label: Speed limit (80km/h)
Match with the Ground Truth :  0.9971826

Correct label: General caution
Prediced label: General caution
Match with the Ground Truth :  0.9989901

Correct label: No entry
Prediced label: No entry
Match with the Ground Truth :  0.99995637

Correct label: Road work
Prediced label: Road work
Match with the Ground Truth :  0.9996406

Correct label: Go straight or right
Prediced label: Go straight or right
Match with the Ground Truth :  0.9972976

Correct label: Yield
Prediced label: Yield
Match with the Ground Truth :  1.0

Analyze Performance

In [45]:
### Calculate the accuracy for these 5 new images. 
### For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate on these new images.

SAVE_DIR = get_dir('SAVE/')
MODEL_SAVE_DIR = get_dir(SAVE_DIR + 'Models/')
MODEL_LOAD_PATH = './' + MODEL_SAVE_DIR + 'densenet.ckpt-29'

tf.reset_default_graph()

with tf.Session() as sess:
    TrafficNet = DenseNetStarter(sess, LEARNING_RATE, WEIGHT_DECAY, KEEP_PROB)
    Saver = tf.train.Saver(max_to_keep=100)
    sess.run(tf.global_variables_initializer())
    if MODEL_LOAD_PATH is not None:
        Saver.restore(sess, MODEL_LOAD_PATH)
        print('Model restored from ' + MODEL_LOAD_PATH)
        acc, loss = TrafficNet.evaluate(im_data_norm, y_data, max_batch = BATCH_SIZE)
        
        print("Final DenseNet accuracy on the test set ", acc*100, "%")
tf.reset_default_graph()
Initalising traffic sign classification model
INFO:tensorflow:Restoring parameters from ./SAVE/Models/densenet.ckpt-29
Model restored from ./SAVE/Models/densenet.ckpt-29
Final DenseNet accuracy on the test set  100.0 %

Output Top 5 Softmax Probabilities For Each Image Found on the Web

For each of the new images, print out the model's softmax probabilities to show the certainty of the model's predictions (limit the output to the top 5 probabilities for each image). tf.nn.top_k could prove helpful here.

The example below demonstrates how tf.nn.top_k can be used to find the top k predictions for each image.

tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.

Take this numpy array as an example. The values in the array represent predictions. The array contains softmax probabilities for five candidate images with six possible classes. tf.nn.top_k is used to choose the three classes with the highest probability:

# (5, 6) array
a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
         0.12789202],
       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
         0.15899337],
       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
         0.23892179],
       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
         0.16505091],
       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
         0.09155967]])

Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:

TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
       [ 0.28086119,  0.27569815,  0.18063401],
       [ 0.26076848,  0.23892179,  0.23664738],
       [ 0.29198961,  0.26234032,  0.16505091],
       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
       [0, 1, 4],
       [0, 5, 1],
       [1, 3, 5],
       [1, 4, 3]], dtype=int32))

Looking just at the first row we get [ 0.34763842, 0.24879643, 0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.

In [ ]:
### Print out the top five softmax probabilities for the predictions on the German traffic sign images found on the web. 
### Feel free to use as many code cells as needed.

Project Writeup

Once you have completed the code implementation, document your results in a project writeup using this template as a guide. The writeup can be in a markdown or pdf file.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.


Step 4 (Optional): Visualize the Neural Network's State with Test Images

This Section is not required to complete but acts as an additional excersise for understaning the output of a neural network's weights. While neural networks can be a great learning device they are often referred to as a black box. We can understand what the weights of a neural network look like better by plotting their feature maps. After successfully training your neural network you can see what it's feature maps look like by plotting the output of the network's weight layers in response to a test stimuli image. From these plotted feature maps, it's possible to see what characteristics of an image the network finds interesting. For a sign, maybe the inner network feature maps react with high activation to the sign's boundary outline or to the contrast in the sign's painted symbol.

Provided for you below is the function code that allows you to get the visualization output of any tensorflow weight layer you want. The inputs to the function should be a stimuli image, one used during training or a new one you provided, and then the tensorflow variable name that represents the layer's state during the training process, for instance if you wanted to see what the LeNet lab's feature maps looked like for it's second convolutional layer you could enter conv2 as the tf_activation variable.

For an example of what feature map outputs look like, check out NVIDIA's results in their paper End-to-End Deep Learning for Self-Driving Cars in the section Visualization of internal CNN State. NVIDIA was able to show that their network's inner weights had high activations to road boundary lines by comparing feature maps from an image with a clear path to one without. Try experimenting with a similar test to show that your trained network's weights are looking for interesting features, whether it's looking at differences in feature maps from images with or without a sign, or even what feature maps look like in a trained network vs a completely untrained one on the same sign image.

Combined Image

Your output should look something like this (above)

In [ ]:
### Visualize your network's feature maps here.
### Feel free to use as many code cells as needed.

# image_input: the test image being fed into the network to produce the feature maps
# tf_activation: should be a tf variable name used during your training procedure that represents the calculated state of a specific weight layer
# activation_min/max: can be used to view the activation contrast in more detail, by default matplot sets min and max to the actual min and max values of the output
# plt_num: used to plot out multiple different weight feature map sets on the same block, just extend the plt number for each new feature map entry

def outputFeatureMap(image_input, tf_activation, activation_min=-1, activation_max=-1 ,plt_num=1):
    # Here make sure to preprocess your image_input in a way your network expects
    # with size, normalization, ect if needed
    # image_input =
    # Note: x should be the same name as your network's tensorflow data placeholder variable
    # If you get an error tf_activation is not defined it may be having trouble accessing the variable from inside a function
    activation = tf_activation.eval(session=sess,feed_dict={x : image_input})
    featuremaps = activation.shape[3]
    plt.figure(plt_num, figsize=(15,15))
    for featuremap in range(featuremaps):
        plt.subplot(6,8, featuremap+1) # sets the number of feature maps to show on each row and column
        plt.title('FeatureMap ' + str(featuremap)) # displays the feature map number
        if activation_min != -1 & activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin =activation_min, vmax=activation_max, cmap="gray")
        elif activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmax=activation_max, cmap="gray")
        elif activation_min !=-1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin=activation_min, cmap="gray")
        else:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", cmap="gray")